from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.021	0.040	0.000	0.002	-1	1	0.629	19.061	0.018	0.629	0.106	0.106
1	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.003	0.000	0.027	-1	1	0.000	0.342	0.007	0.000	0.079	0.079
2	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.994	0.037	0.000	0.003	-1	5	0.688	19.394	0.142	0.688	0.154	0.154
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.028	0.003	0.000	0.028	-1	5	0.000	0.350	0.011	0.000	0.079	0.079
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.373	0.019	0.000	0.002	1	100	0.797	19.147	0.047	0.797	0.124	0.124
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	1	100	0.000	0.371	0.007	0.000	0.061	0.061
6	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.005	0.039	0.000	0.003	-1	100	0.797	19.191	0.040	0.797	0.157	0.157
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.002	0.000	0.027	-1	100	0.000	0.354	0.007	0.000	0.076	0.076
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.372	0.017	0.000	0.002	1	5	0.688	19.351	0.030	0.688	0.123	0.123
9	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	0.000	0.024	1	5	0.000	0.346	0.011	0.000	0.068	0.068
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.403	0.009	0.001	0.001	1	1	0.629	19.183	0.058	0.629	0.073	0.073
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.001	0.000	0.022	1	1	0.000	0.350	0.013	0.000	0.063	0.063
12	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.727	0.025	0.000	0.002	-1	1	0.951	4.424	0.027	0.951	0.390	0.390
13	KNeighborsClassifier_brute_force	predict	100000	1	2	0.011	0.006	0.000	0.011	-1	1	1.000	0.287	0.008	1.000	0.040	0.040
14	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.755	0.023	0.000	0.003	-1	5	0.971	4.421	0.020	0.971	0.623	0.623
15	KNeighborsClassifier_brute_force	predict	100000	1	2	0.008	0.002	0.000	0.008	-1	5	1.000	0.290	0.010	1.000	0.027	0.027
16	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.214	0.010	0.000	0.002	1	100	0.969	4.501	0.073	0.969	0.492	0.492
17	KNeighborsClassifier_brute_force	predict	100000	1	2	0.003	0.000	0.000	0.003	1	100	1.000	0.286	0.007	1.000	0.011	0.011
18	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.777	0.022	0.000	0.003	-1	100	0.969	4.513	0.041	0.969	0.615	0.615
19	KNeighborsClassifier_brute_force	predict	100000	1	2	0.010	0.002	0.000	0.010	-1	100	1.000	0.287	0.007	1.000	0.036	0.036
20	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.206	0.013	0.000	0.002	1	5	0.971	4.442	0.037	0.971	0.497	0.497
21	KNeighborsClassifier_brute_force	predict	100000	1	2	0.003	0.000	0.000	0.003	1	5	1.000	0.288	0.006	1.000	0.011	0.011
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.171	0.005	0.000	0.001	1	1	0.951	4.412	0.032	0.951	0.265	0.265
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.002	0.000	0.000	0.002	1	1	1.000	0.286	0.006	1.000	0.007	0.007

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.664	0.022	0.000	0.004	-1	100	0.907	139.660	0.000	0.907	0.026	0.026
1	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.011	0.001	0.000	0.011	-1	100	1.000	3.138	0.144	1.000	0.004	0.004
2	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	6.280	0.023	0.000	0.006	1	100	0.907	136.554	0.000	0.907	0.046	0.046
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.007	0.001	0.000	0.007	1	100	1.000	3.115	0.078	1.000	0.002	0.002
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.039	0.015	0.000	0.001	1	1	0.868	136.968	0.000	0.868	0.008	0.008
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	0.000	0.002	1	1	1.000	3.135	0.113	1.000	0.001	0.001
6	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.149	0.013	0.000	0.001	-1	5	0.912	139.936	0.000	0.912	0.008	0.008
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	5	1.000	3.119	0.087	1.000	0.001	0.001
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.615	0.006	0.000	0.001	-1	1	0.868	141.751	0.000	0.868	0.004	0.004
9	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.000	0.000	0.004	-1	1	1.000	3.129	0.130	1.000	0.001	0.001
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.982	0.012	0.000	0.002	1	5	0.912	141.842	0.000	0.912	0.014	0.014
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	0.000	0.002	1	5	1.000	3.148	0.097	1.000	0.001	0.001
12	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.050	0.009	0.000	0.000	-1	100	0.910	0.070	0.002	0.910	0.714	0.714
13	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	100	1.000	0.006	0.000	1.000	0.476	0.477
14	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.043	0.001	0.000	0.000	1	100	0.910	0.070	0.003	0.910	0.610	0.611
15	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	100	1.000	0.006	0.000	1.000	0.151	0.151
16	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.029	0.001	0.001	0.000	1	1	0.948	0.049	0.004	0.948	0.598	0.600
17	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	1	1.000	0.006	0.000	1.000	0.133	0.133
18	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.036	0.003	0.000	0.000	-1	5	0.958	0.046	0.001	0.958	0.778	0.778
19	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	5	1.000	0.006	0.000	1.000	0.440	0.442
20	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.033	0.001	0.000	0.000	-1	1	0.948	0.048	0.001	0.948	0.681	0.681
21	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	1	1.000	0.006	0.001	1.000	0.405	0.407
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.031	0.002	0.001	0.000	1	5	0.958	0.046	0.002	0.958	0.673	0.673
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	5	1.000	0.006	0.001	1.000	0.131	0.132

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.164	0.004	300	0.005	0.000	0.796	0.503	0.016	0.796	0.325	0.325
1	HistGradientBoostingClassifier_best	predict	100000	1	100	0.023	0.002	300	0.000	0.023	1.000	0.410	0.012	1.000	0.055	0.055